Physical target movement-mirroring avatar superimposition and visualization system and method in a mixed-reality environment

ABSTRACT

A novel electronic system provides real-time movement-mirroring and three-dimensional (3D) holographic avatar superimposition on a human subject or another physical target located in a holographic mixed-reality environment. In the visual perspectives of a holographic mixed-reality environment viewer wearing a mixed-reality headset or another mixed-reality visualization device, the physical target becomes invisible and is replaced by a 3D holographic avatar in the same coordinates of the physical space where the holographic mixed-reality environment is active. Typically, the 3D holographic avatar is chosen by a system user to represent or “body double” a particular physical target prior to activating the holographic mixed-reality environment. Once activated, the novel electronic system provides subject feature extraction, subject-to-avatar recognition, subject pose and expression matching, motion retargeting, and movement mirroring to reflect detailed movements and facial or bodily expressions of the physical target in the 3D holographic avatar in real time.

BACKGROUND OF THE INVENTION

The present invention generally relates to mixed and immersivethree-dimensional (3D) synthetic computer-graphic objects and physicalobjects visualizations in a mixed-reality environment. In particular,the present invention relates to physical target movement-mirroring 3Davatar holographic superimposition and visualization systems and relatedmethods of operations in a mixed-reality environment. Furthermore, thepresent invention also relates to computer graphics generation of 3Davatars as superimposed real-time motion representations of targetedhuman subjects in a holographic mixed-reality (HMR) live environment formixed-reality viewers. In addition, the present invention also relatesto immersive mixed-reality visualization of holographic 3D avatars that“envelope” corresponding targeted human subjects and real physicalobjects in the same physical space of the HMR live environment.

In recent years, virtual reality (VR) and augmented reality (AR)applications are gaining increasing popularity and relevance inelectronic user applications. For example, VR headsets for computers andportable devices are able to provide interactive and stereoscopic gamingexperiences, training simulations, and educational environments forusers wearing the VR headsets. In another example, augmented reality(AR) mobile applications are designed to add texts, descriptions, oradded (i.e. “augmented”) digitized materials to physical objects if auser wears AR goggles or utilizes AR-compatible mobile applicationsexecuted in portable devices. For one of ordinary skill in the art,virtual reality (VR) refers to a completely computer-generated syntheticenvironment with no direct correlations to a real physical space or areal physical object, while augmented reality (AR) refers to descriptivedigital materials that are displayed next to a machine-recognized realphysical object to add or “augment” more information to the physicalreality.

Nevertheless, conventional VR and AR applications are unable to provideseamless integration of ultra-high resolution and lifelike holographicthree-dimensional objects that can be juxtaposed or intermixed with realphysical objects in the same physical location for interactive andimmersive mixed experiences, because the conventional VR applicationsmerely provide user interactions in a purely computer-generatedsynthetic (i.e. virtual) environment with no correlation to physicalobjects in a real physical space, while the conventional AR applicationsmerely provide additional informational overlays (i.e. informationaugmentation) to machine-recognized real physical objects viapartially-transparent AR goggles or AR-enabled camera applications inmobile devices.

A recent evolution of conventional VR and AR applications has resultedin an innovative intermixture of computer-generated lifelike holographicobjects and real objects that are synchronized and correlated to aparticular physical space (i.e. as a “mixed-reality” (MR) environment)for immersive user interactions during the user's visit to theparticular physical space. However, the mixed-reality applications inthe consumer electronics market today are primarily focused on displayand interactions between synthetically-created holographic objects andphysical objects. For example, in existing MR applications, computergraphics-created synthetic cartoon holograms may be positioned next to aphysical painting on a physical wall, and be viewable simultaneouslythrough a mixed-reality environment viewing headset unit. In suchconventional mixed-reality environment creation layouts, holograms arecreated in a computer graphics server first in isolation from physicalobjects in the same physical space. Then, reference coordinates areutilized at a later timeframe to correlate the position of the hologramsto the physical objects in the same physical space for intermixeddisplay of the holograms and the physical objects through a user'sMR-viewing headset unit.

Presently, the mixed-reality display systems in the market today areunable to provide more advanced levels of interactivity andvisualizations among holograms and physical objects. For example,conventional mixed-reality display systems are not designed to orcapable of providing subject motion feedback-based dynamic rendering ofholograms for display in real time within a mixed-reality environment.Likewise, conventional mixed-reality display systems are unable toaccommodate virtualized switchover roles and images between hologramsand physical objects in the mixed-reality environment. These types ofadvanced levels of interactivity and visualizations, if made possible,may increase the appeal of widespread deployment of mixed-realityapplications for wearable display devices and other portable electronicdevices by enhancing mixed-reality environment design flexibility andimmersive contents in related mixed-reality applications.

Therefore, it may be advantageous to provide a novel electronic systemand a related method of operation that enable more advanced levels ofinteractivity and visualizations in a mixed-reality environment, such asvirtualized switchover roles and images between holograms and physicalobjects, and dynamic real-time rendering of holographic motionssuperimposed on targeted physical objects.

Furthermore, it may also be advantageous to provide a novel electronicsystem and a related method of operation that accommodate physicaltarget movement-mirroring avatar superimposition and visualization in amixed-reality environment for enhanced immersive mixed-reality contentsand mixed-reality choreographic scenario design flexibilities.

In addition, it may also be advantageous to provide a novel electronicsystem that accommodates a mixed-reality system user to create, select,or modify a preferred 3D holographic avatar as a holographicallymotion-mirrored and fully encapsulated computerized visualrepresentation of herself or himself for real-time viewing by otherusers immersed in a mixed-reality environment.

SUMMARY

Summary and Abstract summarize some aspects of the present invention.Simplifications or omissions may have been made to avoid obscuring thepurpose of the Summary or the Abstract. These simplifications oromissions are not intended to limit the scope of the present invention.

In one embodiment of the invention, a method for creating physicaltarget movement-mirroring three-dimensional (3D) holographic avatarsuperimposition and visualization in a mixed-reality environment isdisclosed. This method comprises the steps of: (1) choosing, via asystem user application interface, a 3D holographic avatar that visuallyencapsulates a physical target located in the mixed-reality environment,which is generated by a physical target movement-mirroring 3Dholographic avatar superimposition and visualization creation system;(2) recognizing and correlating the physical target with the 3Dholographic avatar by capturing an image of the physical target in acamera of a mixed-reality headset worn by a holographic mixed-reality(HMR) viewer, extracting graphical feature points from the image,comparing the graphical feature points to various 3D holographic avatarsstored in a physical target movement-mirroring avatar database, andfinding a correct match for a graphical similarity between the physicaltarget and the 3D holographic avatar previously chosen by a user toencapsulate the physical target; (3) tracking motions and poses of thephysical target via a continuous image capturing from the camera on themixed-reality headset and a real-time extraction of moving graphicalfeature points of the physical target; (4) retargeting the motions andthe poses of the physical target to the 3D holographic avatar bymatching tracking points of similar body parts and facial featuresbetween the physical target and the 3D holographic avatar, and byexecuting a deep-learning pose estimation engine that retrieves amatching pose feature from a pre-defined pose feature database for 3Dholographic avatars; (5) continuously matching the tracking points ofsimilar body parts and facial features between the physical target andthe 3D holographic avatar in real time as long as the physical target ismaking a movement or a change in facial or bodily expressions whilebeing present in the mixed-reality environment; (6) continuouslyexecuting motion retargeting, in real time, from the physical target tothe 3D holographic avatar to mirror and mimic the movement or the changein facial or bodily expressions; and (7) from a visual perspective ofthe HMR viewer wearing the mixed-reality headset or another viewingdevice, completely encapsulating the physical target to replace anaked-eye view of the physical target with the 3D holographic avatarthat also mirrors and mimics the motions and the poses of the physicaltarget in real time, whenever the mixed-reality environment is active.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows three-dimensional (3D) holographic avatars envelopingphysical targets in a mixed-reality environment generated by a physicaltarget movement-mirroring avatar superimposition and visualizationcreation system, in accordance with an embodiment of the invention.

FIG. 2 shows an instance of physical target movement-mirroring avatarsuperimposition and visualization in a mixed-reality environmentprovided by the novel electronic system, in accordance with anembodiment of the invention.

FIG. 3 shows first two steps in creating physical targetmovement-mirroring avatar superimposition and visualization in amixed-reality environment by the novel electronic system, in accordancewith an embodiment of the invention.

FIG. 4 shows third and fourth steps in creating physical targetmovement-mirroring avatar superimposition and visualization in amixed-reality environment by the novel electronic system, in accordancewith an embodiment of the invention.

FIG. 5 shows fifth and sixth steps in creating physical targetmovement-mirroring avatar superimposition and visualization in amixed-reality environment by the novel electronic system, in accordancewith an embodiment of the invention.

FIG. 6 shows a system block diagram for physical targetmovement-mirroring avatar superimposition and visualization creationsystem in a mixed-reality environment, in accordance with an embodimentof the invention.

DETAILED DESCRIPTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

In the following detailed description of embodiments of the invention,numerous specific details are set forth in order to provide a morethorough understanding of the invention. However, it will be apparent toone of ordinary skill in the art that the invention may be practicedwithout these specific details. In other instances, well-known featureshave not been described in detail to avoid unnecessarily complicatingthe description.

The detailed description is presented largely in terms of description ofshapes, configurations, and/or other symbolic representations thatdirectly or indirectly resemble one or more electronic systems andmethods for physical target movement-mirroring avatar superimpositionand visualization creation in a mixed-reality environment. These processdescriptions and representations are the means used by those experiencedor skilled in the art to most effectively convey the substance of theirwork to others skilled in the art.

Reference herein to “one embodiment” or “an embodiment” means that aparticular feature, structure, or characteristic described in connectionwith the embodiment can be included in at least one embodiment of theinvention. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment. Furthermore, separate or alternative embodiments arenot necessarily mutually exclusive of other embodiments. Moreover, theorder of blocks in process flowcharts or diagrams representing one ormore embodiments of the invention does not inherently indicate anyparticular order and do not imply any limitations in the invention.

One objective of an embodiment of the present invention is to provide anovel electronic system and a related method of operation that enablemore advanced levels of interactivity and visualizations in amixed-reality environment. Examples of such advanced levels ofinteractivity and visualizations in a mixed-reality environment include,but are not limited to, virtualized switchover roles and images betweenholograms and physical objects, and dynamic real-time rendering ofholographic motions superimposed on targeted physical objects.

Furthermore, another objective of an embodiment of the invention is toprovide a novel electronic system and a related method of operation thataccommodate physical target movement-mirroring avatar superimpositionand visualization in a mixed-reality environment for enhanced immersivemixed-reality contents and mixed-reality choreographic scenario designflexibilities.

Another objective of an embodiment of the present invention is toprovide a novel electronic system that empowers a mixed-reality systemuser with electronic user interfaces, physical target movement-mirroringavatar superimposition user apps and avatar databases, and apprepository developer tools to create, select, or modify a preferred 3Dholographic avatar as a holographically motion-mirrored and fullyenveloped or encapsulated computerized visual representation of herselfor himself for real-time viewing by other users immersed in amixed-reality environment.

For the purpose of describing the invention, a term referred to as“mixed reality,” or “MR,” as an acronym, is defined as an intermixtureof computer-generated lifelike holographic objects and real physicalobjects that are synchronized and correlated to a particular physicalspace for immersive user interactions during the user's visit to theparticular physical space. Typically, unlike conventional augmentedreality applications, the computer-generated lifelike holographicobjects are ultra high-resolution (e.g. 4K/UHD) or high-resolution (e.g.HD quality or above) three-dimensional synthetic objects that areintermixed and/or juxtaposed to real physical objects, wherein a viewerimmersed in the mixed-reality environment is often unable to distinguishthe synthetic nature of the computer-generated lifelike holographicobjects and the real physical objects provided by the mixed-realityenvironment.

The viewer immersed in the mixed-reality environment may be locallypresent at the particular physical space correlated and synchronizedwith the computer-generated lifelike holographic objects and the realphysical objects in one or more mixed-reality artificial layerssuperimposed on the particular physical space. Alternatively, the viewermay also be remotely located in a different physical space but stillcorrelated and synchronized with the particular physical space to beimmersed in a holographic mixed-reality (HMR) environment that providesthe computer-generated lifelike holographic objects, wherein the HMRenvironment is synthesized and guided in real time through amixed-reality recording headset worn by an onsite surrogate visitor tothe particular physical space. In the alternate embodiment of theinvention, the remotely-located viewer (i.e. a remote visitor) is alsorequired to wear a head-mounted display (HMD) device or at least utilizea mobile electronic device configured to execute a mixed-reality mobileapplication, in order to experience the holographic mixed-reality (HMR)environment streaming from a physical target movement-mirroring avatarsuperimposition and visualization creation system.

Moreover, for the purpose of describing the invention, a term referredto as “avatar” is defined as a three-dimensional (3D) model or ahologram that represents or symbolizes a physical target, which is alsopresent in a holographic mixed-reality (HMR) environment. In a preferredembodiment of the invention, avatars may be humanized figures, cartoonfigures, animals, or nonlife objects. In addition, examples of “physicaltargets” that can be represented by avatars include, but are not limitedto, a human subject, an animal, and an inanimate object (e.g. apainting, a sculpture, a piece of furniture, etc.) that are physicallyand originally present in the HMR environment as real objects, and notas holograms.

In addition, for the purpose of describing the invention, the words“envelope,” “encapsulate,” or any other verb, adjective, or adverbvariation of these two words refer to a computer graphics-basedtransformation of a real image of a physical target into a holographicavatar of a user's choice with real-time motion mirroring, so that anymovements and gestures of the real image are entirely mimicked andreflected as those of the holographic avatar in real time, when viewedthrough a head-mounted display (HMD) or another portable electronicdevice that provides a holographic mixed-reality (HMR) environment. In apreferred embodiment of the invention, a physical target that undergoesa user-chosen avatar envelopment or encapsulation in the HMR environmentwill appear as if the physical target is fully “enveloped” or“encapsulated” in a holographic “jumpsuit,” thus making the user-chosenavatar a real-time motion-mirrored representation of the physicaltarget, while the physical target itself becomes invisible in the eyesof the HMR viewers who wear headsets or other mixed-reality viewingequipment, as illustrated in FIGS. 1-2.

Furthermore, for the purpose of describing the invention, a termreferred to as “HoloWalks” is defined as a novel electronic system thatprovides, executes, enables, and manages a three-dimensional (3D)mixed-reality (MR) space with at least one MR artificial layersuperimposed on a physical space, a mixed-reality (MR) experienceconstruction conceived by an MR experience designer (i.e. a userinteraction choreography designer), and a 3D MR experience sharing withtourists, visitors, and other users who visit the physical space whilewearing a head-mounted display device or utilizing an MR-enabled mobileapplication executed on a mobile device.

In addition, for the purpose of describing the invention, a termreferred to as a “mixed-reality artificial layer” is defined as acomputer-generated graphics layer in which mixed-reality objects (MROs)and/or mixed-reality holographic avatars are created and positioned by aphysical target movement-mirroring avatar superimposition andvisualization creation system onto virtual coordinates, which correlateto a particular physical space of a viewer's interest, such as a workcollaborative room, a concert hall, a museum, an exhibition venue, alecture hall, a research facility, or a tourist destination.

Moreover, for the purpose of describing the invention, a term referredto as “hologram” is defined as a three-dimensional holographic objectconfigured to be displayed from a head-mounted display (HMD) device, amobile device executing a mixed-reality visual mobile application, oranother electronic device with a visual display unit. Typically, ahologram is capable of being animated as a three-dimensional elementover a defined period of time. Examples of holograms utilized inmixed-reality environments generated by a physical targetmovement-mirroring avatar superimposition and visualization creationsystem include, but are not limited to, a cartoon avatar, a humanizedavatar, a mixed-reality object (MRO), or another mixed-reality hologram,which can be intermixed with or juxtaposed to physical objects forseamlessly-vivid visualizations of both artificial holograms andphysical objects.

In addition, for the purpose of describing the invention, a termreferred to as “three-dimensional model,” or “3D model,” is defined asone or more computer-generated three-dimensional avatars, images,videos, or holograms. In a preferred embodiment of the invention, acomputerized 3D model is created as a hologram after multi-angle videodata are extracted, transformed, and reconstructed by three-dimensionalgraphics processing algorithms executed in a computer system or in acloud computing resource comprising a plurality of networked andparallel-processing computer systems. The computer-generated 3D modelcan then be utilized as a mixed-reality object (MRO) or a humanizedmixed-reality hologram (MRH) in a mixed-reality artificial layersuperimposed on a particular physical space correlated by virtualcoordinates from a physical target movement-mirroring avatarsuperimposition and visualization creation system.

Moreover, for the purpose of describing the invention, a term referredto as “cloud” is defined as a scalable data network-connected and/orparallel-processing environment for complex graphics computations,transformations, and processing. The data network-connected and/orparallel-processing environment can be provided using a physicalconnection, a wireless connection, or both. For example, a cloudcomputing resource comprising a first cloud computing server, a secondcloud computing server, and/or any additional number of cloud computingservers can each extract and transform a portion of multi-angle videodata simultaneously as part of a scalable parallel processing algorithm,which performs temporal, spatial, and photometrical calibrations, andexecutes depth map computation, voxel grid reconstruction, and deformedmesh generation. A scalable number of cloud computing servers enables areal-time or near real-time transformation and reconstruction of 3Dmodels after consumer video recording devices transmit multi-angle videodata to the cloud computing resource.

Furthermore, for the purpose of describing the invention, a termreferred to as “HoloPortal” is defined as a 3D model creation studiothat incorporates cameras positioned on a multiple number of anglesaround a stage, where a target object is placed for video footagerecording at the multiple number of angles around the stage. The 3Dmodel creation studio also typically incorporates a real-time or nearreal-time 3D reconstruction electronic system, which is configured toperform silhouette extractions, 3D voxel generation, 3D mesh generation,and texture and detail-adding operations to create a user-controllablethree-dimensional model that resembles the target object.

In addition, for the purpose of describing the invention, a termreferred to as “HoloCloud” is defined as a novel electronic system thatcaptures live multi-angle video feeds of a target object with portableelectronic devices and generates a user-controllable three-dimensionalmodel by performing various 3D reconstruction calculations andprocedures in a scalable cloud computing network. In one example, aHoloCloud system comprises a plurality of common consumer-level videorecording devices (e.g. smartphones, camcorders, digital cameras, etc.)positioned in various angles surrounding a target object (e.g. a human,an animal, a moving object, etc.), a scalable number of graphicprocessing units (GPU's) in a scalable cloud computing platform, a 3Dpre-processing module, a 3D reconstruction module, a background 3Dgraphics content, a 360-degree virtual reality or video content, and adynamic 3D model created by the 3D reconstruction module. In oneembodiment, the 3D pre-processing module and the 3D reconstructionmodules are graphics processing software executed in the scalable numberof graphic processing units (GPU's). In another embodiment, thesemodules may be hard-coded specialized semiconductor chipsets or anotherhardware that operate in conjunction with the GPU's to provide 3Dprocessing and reconstruction.

FIG. 1 shows three-dimensional (3D) holographic avatars (102B, 103B)enveloping physical targets (102A, 103A) in a mixed-reality environment(100) generated by a physical target movement-mirroring avatarsuperimposition and visualization creation system, in accordance with anembodiment of the invention. In this illustration, a first physicaltarget (102A) is a first human participant and a second physical target(103A) is a second human participant present in a physical space, whichis part of the mixed-reality environment (100) of FIG. 1.

When a holographic mixed-reality (HMR) viewer (101) wears amixed-reality headset (104) or utilizes another electronic equipment toexperience the mixed-reality environment (100), as shown in FIG. 1, theHMR viewer (101) sees a first 3D holographic avatar (102B) encapsulatingand motion-mirroring the first physical target (102A), and a second 3Dholographic avatar (103B) encapsulating and motion-mirroring the secondphysical target (103A) in real time. If a physical targetmotion-mirrored by a 3D holographic avatar is a human or an animal, thephysical target movement-mirroring avatar superimposition andvisualization creation system tracks the physical target's movements aswell as facial expressions, and then provides novel graphics processingsteps on the fly to ensure that the 3D holographic avatar mimics all ofthe physical target's motions and expressions in real time

For example, if a human participant designated as the second physicaltarget (103A) in FIG. 1 smiles or winks, the second 3D holographicavatar (103B) that encapsulates the human participant in themixed-reality environment (100) also smiles or winks to mirror the humanparticipant in real time. In this example, the second 3D holographicavatar (103B) is preferably a full life-size 3D avatar capable of rapidand natural motions that directly mirror, correlate, or reflect theunderlying motions of the second physical target (103A). Preferably, thephysical target movement-mirroring avatar superimposition andvisualization creation system incorporates a scalable number of highperformance computer servers and graphics processing units to enable thereal-time physical target movement-mirroring and avatar superimpositionby taking several graphics-intensive processing steps.

In one instance of an embodiment of the invention, the physical targetmovement-mirroring avatar superimposition and visualization steps mayinvolve the following multiple steps: (1) recognizing and correlating aparticular physical target with a chosen 3D avatar previouslysynthesized or selected by a system user; (2) tracking motions and posesfrom the particular physical target and retargeting such motions andposes to the chosen 3D avatar based on deep-learning pose estimations;(3) matching extracted features or tracking points from the particularphysical target with the chosen 3D avatar in real time; (4) executingmotion retargeting by correlating the real-time motion tracking pointsof the particular physical target to the chosen 3D avatar to mirror andmimic the movements and the expressions of the particular physicaltarget in the chosen 3D avatar; (5) completely enveloping orencapsulating the particular physical target to replace the naked-eyeview of the particular physical target with the chosen 3D avatar thatalso mirrors and mimics the underlying motions and the expressions ofthe particular physical target, whenever the system viewer is immersedin a mixed-reality environment through a mixed-reality headset gear oranother viewing device.

In a preferred embodiment of the invention, a specific 3D holographicavatar (e.g. 102B, 103B) is selected by a system user (e.g. 101, 102A,or 103A) to represent a particular physical target through a system userinterface. Once selected and activated to represent the particularphysical target, the specific 3D holographic avatar is configured toencapsulate or envelope the particular physical target completely, thusmaking the particular physical target invisible from the HMR viewer's(101) perspective. The invisible physical target is visually andelectronically replaced and motion-mirrored with the specific 3Dholographic avatar while the mixed-reality environment (100) is active.Furthermore, as also shown in FIG. 1, the physical targets (102A, 103A)are actually present in the physical space and certainly visible tonaked-eyes viewers who are not wearing or utilizing mixed-realityviewing equipment.

FIG. 2 shows an instance of physical target movement-mirroring avatarsuperimposition and visualization in a mixed-reality environment (200)provided by the novel electronic system, in accordance with anembodiment of the invention. As shown in this figure, the first physicaltarget (i.e. 102A in FIG. 1) is now visually invisible and completelyenveloped by the first 3D holographic avatar (102B), which encapsulatesand motion-mirrors the first physical target (i.e. 102A in FIG. 1) inreal time, when viewed in the mixed-reality environment (200) by theholographic mixed-reality (HMR) viewer (101). Similarly, the secondphysical target (i.e. 103A in FIG. 1) is now also visually invisible andcompletely enveloped by the second 3D holographic avatar (103B), whichencapsulates and motion-mirrors the second physical target (i.e. 103A inFIG. 1) in real time, when viewed in the mixed-reality environment (200)by the holographic mixed-reality (HMR) viewer (101).

It should be noted that in the mixed-reality environment (200) asillustrated in FIG. 2, all of the physical targets (i.e. 102A, 103A inFIG. 1) as well as the HMR viewer (101) are still physically present inthe physical space, but in the eyes of the HMR viewer (101) utilizingthe mixed-reality headset (104) to immerse in the mixed-realityenvironment (200), the physical targets have transformed into life-sized3D avatars that also mirror and reflect the underlying physical targets'movements and expressions in real time. In the preferred embodiment ofthe invention, each 3D holographic avatar completely envelopes a chosenphysical target, as if the chosen physical target is wearing aholographic jumpsuit that also tracks and mimics the chosen physicaltarget's movements and expressions. In particular, a 3D holographicavatar that visually encapsulates a physical target located in themixed-reality environment serves the novel role of a full-bodyholographic jumpsuit that makes the physical target invisible whileretaining real-time motions and expressions originating from thephysical target, in the visual perspective of an HMR viewer wearing amixed-reality headset or another viewing device.

If a physical target motion-mirrored by a 3D holographic avatar is ahuman or an animal, the physical target movement-mirroring avatarsuperimposition and visualization creation system tracks the physicaltarget's movements as well as facial expressions, and then providesnovel graphics processing steps on the fly to ensure that the 3Dholographic avatar mimics all of the physical target's motions andexpressions in real time. For example, if a human participant designatedas a physical target waves and then puts his or her thumbs up, then theuser-selected 3D holographic avatar that encapsulates the humanparticipant in the mixed-reality environment (200) also waves and putsthumbs up to mirror the human participant in real time. Theuser-selected 3D holographic avatar is preferably a full life-size 3Davatar capable of rapid and natural motions that directly mirror,correlate, or reflect the underlying motions of the physical target.

In the preferred embodiment of the invention, the physical targetmovement-mirroring avatar superimposition and visualization creationsystem incorporates a scalable number of high performance computerservers and graphics processing units to enable the real-time physicaltarget movement-mirroring and avatar superimposition by taking severalgraphics-intensive processing steps. In one instance of an embodiment ofthe invention, the physical target movement-mirroring avatarsuperimposition and visualization steps may involve the followingmultiple steps: (1) recognizing and correlating a particular physicaltarget with a chosen 3D avatar previously synthesized or selected by asystem user; (2) tracking motions and poses from the particular physicaltarget and retargeting such motions and poses to the chosen 3D avatarbased on deep-learning pose estimations; (3) matching extracted featuresor tracking points from the particular physical target with the chosen3D avatar in real time; (4) executing motion retargeting by correlatingthe real-time motion tracking points of the particular physical targetto the chosen 3D avatar to mirror and mimic the movements and theexpressions of the particular physical target in the chosen 3D avatar;(5) completely enveloping or encapsulating the particular physicaltarget to replace the naked-eye view of the particular physical targetwith the chosen 3D avatar that also mirrors and mimics the underlyingmotions and the expressions of the particular physical target, wheneverthe system viewer is immersed in a mixed-reality environment through amixed-reality headset gear or another viewing device.

In the illustrative example as shown in FIG. 2, a specific 3Dholographic avatar (e.g. 102B or 103B) is selected by a system user(e.g. 101, 102A, or 103A in FIG. 1) to represent a particular physicaltarget through a system user interface. Once selected and activated torepresent the particular physical target, the specific 3D holographicavatar is configured to encapsulate or envelope the particular physicaltarget completely, thus making the particular physical target invisiblefrom the HMR viewer's (101) perspective. The invisible physical targetis visually and electronically replaced and motion-mirrored with thespecific 3D holographic avatar while the mixed-reality environment (200)is active. Furthermore, as previously shown in FIG. 1, the physicaltargets (102A, 103A) are actually present in the physical space andcertainly visible to naked-eyes viewers who are not wearing or utilizingmixed-reality viewing equipment.

FIG. 3 shows first two steps (i.e. STEP 301 and STEP 302) in creatingphysical target movement-mirroring avatar superimposition andvisualization in a mixed-reality environment (300) by the novelelectronic system, in accordance with an embodiment of the invention. InSTEP 301 as shown in FIG. 3, a system user (e.g. 307A, 308A, or 309)selects a 3D holographic avatar (e.g. 307B or 308B) to represent andsubstitute a particular physical object present in a physical space withthe 3D holographic avatar, where the mixed-reality environment (300) isoperated and provided by the physical target movement-mirroring avatarsuperimposition and visualization creation system. The particularphysical object present in the physical space may be a human subject(e.g. 307A or 308A), an animal, or an inanimate object such as apainting, a piece of furniture, or another tangible item. Furthermore,the system user that selects the 3D holographic avatar (e.g. 307B or308B) to substitute the particular physical object with the 3Dholographic avatar may be the human subject (e.g. 307A or 308A) himselfor herself in some instances, or a holographic mixed-reality (HMR)viewer (309) wearing a mixed-reality headset (310) in other instances.

In the example shown in STEP 301, a first 3D holographic avatar (307B)represents a first human subject (307A) with real-time motion and facialexpression mirroring. Likewise, a second 3D holographic avatar (308B)represents a second human subject (308A) with real-time motion andfacial expression mirroring. The physical target movement-mirroringavatar superimposition and visualization creation system includes arobust database of 3D holographic avatars created by third-partydevelopers, graphics artists, or system users themselves. Furthermore,in the preferred embodiment of the invention, the real-time motion andfacial expression mirroring achieve optimal computational efficienciesby utilizing an artificial intelligence (AI) and deep learning-basedpose estimation engine for 3D holographic avatars and pre-defined posefeature databases for avatars, which in turn reduce graphics processingdelay and computational burden in providing real-time motion and facialexpression mirroring.

The second step in creating physical target movement-mirroring avatarsuperimposition and visualization in the mixed-reality environment (300)involves a machine-determined recognition of the first human subject(307A) and the second human subject (308A) through the mixed-realityheadset (310) and/or cameras worn by the holographic mixed-reality (HMR)viewer (309), as shown in STEP 302. In one embodiment of the invention,based on user selection of 3D holographic avatars to complete visualencapsulation of the human subjects (307A, 308A) from STEP 301, thephysical target movement-mirroring avatar superimposition andvisualization creation system performs facial recognition algorithms onthe subjects' faces (311, 312) in a subject feature extraction block(e.g. 607 in FIG. 6) and a subject-to-avatar recognition and posetracking block (e.g. 605 in FIG. 6) to extract facial feature points,which are then utilized to determine, match, and retrieve theuser-selected 3D holographic avatars (307B, 308B) that are intended toencapsulate the human subjects (307A, 308A) when viewed in themixed-reality environment (300). In other embodiments of the invention,the matching determination and the retrieval of the user-selected 3Dholographic avatars associated with a human subject or another realobject present in a physical space may be based on gait, Bluetooth IDs,or other object identification methods.

In case of the illustrative example as shown in STEP 302 in FIG. 3, thesubject-to-avatar recognition and pose tracking block (e.g. 605 in FIG.6) in the physical target movement-mirroring avatar superimposition andvisualization creation system internally identifies the first humansubject (307A) as “USER 2020,” and the second human subject (308A) as“USER 4045.” In this example, the system's internal user identificationdesignations are dynamically linked to the user-selected 3D holographicavatars that will act as full holographic body jumpsuits to theidentified human subjects. Therefore, the system's determination of thefirst human subject (307A) as “User 2020” from its facial recognitionprocess retrieves the graphics dataset for “User 2020-prime” (User2020′), which in this case is the first 3D holographic avatar (307B)previously selected by the user in STEP 301 from a holographic avatardatabase. Likewise, the system's determination of the second humansubject (308A) as “User 4045” from its facial recognition processretrieves the graphics dataset for “User 4045-prime” (User 4045′), whichin this case is the second 3D holographic avatar (308B) previouslyselected by the user in STEP 301 from a holographic avatar database(e.g. 608 in FIG. 6). The holographic avatar database may also beoperatively connected to app repository developer tools (e.g. 609 inFIG. 6) that enable a plurality of internal or third-party developers toprovide various physical target movement-mirroring avatarsuperimposition desktop or mobile apps and a diverse set of 3D graphicalavatars for the user to choose from.

FIG. 4 shows third and fourth steps (i.e. STEP 303 and STEP 304) increating physical target movement-mirroring avatar superimposition andvisualization in a mixed-reality environment (400) by the novelelectronic system, in accordance with an embodiment of the invention. Inthe third step as shown in STEP 303, the physical targetmovement-mirroring avatar superimposition and visualization creationsystem is configured to track and recognize motions and poses of eachhuman subject (i.e. 307A and 308B) by extracting graphical featurepoints of each subject's motions and poses in real time. The extractedgraphical feature points are then inputted into a deep-learningartificial intelligence (AI) pose estimation engine (e.g. 610 in FIG. 6)for 3D holographic avatars to synthesize machine-determined pre-definedpose features and estimations to mirror the human subjects' motions andposes in the corresponding user-chosen 3D holographic avatars.

The newly-synthesized machine-determined pre-defined pose features andestimations are then further utilized in the fourth step by the physicaltarget movement-mirroring avatar superimposition and visualizationcreation system to retarget and update motions and poses of auser-chosen 3D holographic avatar (e.g. 308B) to mirror a correspondinghuman subject (e.g. 308A), as shown in STEP 304. In the preferredembodiment of the invention, the human subject's (e.g. 308A) face, body,arms, and legs are dynamically tracked in real time with multipletracking points (401, 402, 403, 404), which are assigned by asubject-to-avatar pose matching and real-time movement mirroring andretargeting engine (e.g. 603 in FIG. 6) of the physical targetmovement-mirroring avatar superimposition and visualization creationsystem.

In the preferred embodiment of the invention, the multiple trackingpoints (401, 402, 403, 404) enable real-time tracking of the detailedbodily and facial movements through continuous feature extractions ofthe human subject by the physical target movement-mirroring avatarsuperimposition and visualization creation system. The continuousfeature extractions of the human subject are then matched to theuser-chosen 3D holographic avatar (e.g. 308B) to invoke or createcorresponding visual changes in the facial and the bodily expressions ofthe user-chosen 3D holographic avatar in real time, without perceptibledelay from the eyes of the holographic mixed-reality (HMR) viewer (e.g.309).

FIG. 5 shows fifth and sixth steps (i.e. STEP 305 and STEP 306) increating physical target movement-mirroring avatar superimposition andvisualization in a mixed-reality environment (500) by the novelelectronic system, in accordance with an embodiment of the invention. Inthe fifth step as shown in STEP 305, the subject-to-avatar pose matchingand real-time movement mirroring and retargeting engine (e.g. 603 inFIG. 6) of the physical target movement-mirroring avatar superimpositionand visualization creation system executes motion retargeting bycorrelating the real-time motion tracking points of the human subject(e.g. 308A in FIG. 4) to the user-chosen 3D holographic avatar (e.g.308B) to mirror and mimic the movements and the expressions of the humansubject in the user-chosen 3D holographic avatar.

In the preferred embodiment of the invention, the motion retargeting andmirroring performed by the subject-to-avatar pose matching and real-timemovement mirroring and retargeting engine achieves additional efficiencyand processing speed by querying a pre-defined pose feature database foravatars (e.g. 611 in FIG. 6) and rapidly determining the mostappropriate changes in the avatar's pose based on the outputrecommendations from the deep-learning artificial intelligence (AI) poseestimation engine (e.g. 610 in FIG. 6) for 3D holographic avatars. Whenthe human subject (e.g. 308A in FIG. 4) or another tracked physicaltarget moves or changes expressions, the changes in pose and motionextraction point data for the tracked physical target invoke a similarmagnitude of pose and motion changes in the user-chosen 3D holographicavatar (e.g. 308B). Such reflective and corresponding changes in motionand expressions in the user-chosen 3D holographic avatar fundamentally“mirror” whatever the underlying physical target is doing, while thephysical target still remains invisible and totally encapsulated by theuser-chosen 3D holographic avatar in the eyes of the holographicmixed-reality (HMR) viewer.

Once the subject-to-avatar pose matching and real-time movementmirroring and retargeting engine (e.g. 603 in FIG. 6) of the physicaltarget movement-mirroring avatar superimposition and visualizationcreation system completes the motion retargeting synthesis for theuser-chosen 3D holographic avatar (e.g. 308B), a subjectmovement-mirroring avatar and motion-retargeting 3D hologram imagegenerator (e.g. 604 in FIG. 6) utilizes the synthesized retargetedmotion data to generate one or more updated motion frames (501A, 501B,501C) for the user-chosen 3D holographic avatar, as shown in STEP 306 inFIG. 5. Then, an avatar visualizer block (e.g. 606 in FIG. 6) in thephysical target movement-mirroring avatar superimposition andvisualization creation system can display the updated motion frames(501A, 501B, 501C) for the user-chosen 3D holographic avatar in the eyesof the holographic mixed-reality (HMR) viewer, when the mixed-realityenvironment (500) is active. When a human subject or another physicaltarget moves, the user-chosen 3D holographic avatar also moves withinthe mixed-reality physical space in equal direction and magnitude tomirror the physical target's changes in motion and location, whilekeeping the underlying physical target encapsulated and invisible to theHMR viewer.

As illustrated and described in conjunction with FIGS. 3-5, theresulting effect of the physical target movement-mirroring avatarsuperimposition and visualization creation steps is a novelmixed-reality environment in which a holographic mixed-reality (HMR)viewer visualizes and interacts with a user-chosen 3D avatar thatmirrors and mimics the underlying motions and the expressions of aphysical target also present in the mixed-reality space, while thephysical target itself remains invisible to the HMR viewer wearing aheadset gear or another mixed-reality viewing device.

FIG. 6 shows a system block diagram (600) for the physical targetmovement-mirroring avatar superimposition and visualization creationsystem in a mixed-reality environment, in accordance with an embodimentof the invention. In this example, the system incorporates amixed-reality (MR) environment graphics generation hardware (H/W)infrastructure (601), which comprises graphics processing units (GPUs),memory units, non-volatile data storages, graphics servers incorporatingGPUs for additional computations and network communications,cloud-networked scalable computing resources, and/or other hardwarecomponents for 3D hologram generation, motion retargeting, andvisualization in the MR environment.

Furthermore, as shown in the system block diagram (600), the physicaltarget movement-mirroring avatar superimposition and visualizationcreation system also includes a mixed-reality (MR) environment graphicsgeneration and holographic 3D visualization operating system (602),which is executed in the GPUs and the memory units of the MR environmentgraphics generation H/W infrastructure (601). This operating systemserves as a foundation software operation layer for mixed-realityscenario design capabilities, subject-to-avatar recognition,subject-to-avatar real-time motion-retargeting, and holographicvisualization tasks that are further enabled by novel application layersand programs (e.g. 603, 604, 605, 606, 607, 608, 609, 610, 611), whichare dynamically executed on top of the mixed-reality (MR) environmentgraphics generation and holographic 3D visualization operating system(602).

The physical target movement-mirroring avatar superimposition andvisualization creation system further comprises a subject-to-avatar posematching and real-time movement mirroring and retargeting engine (603),a subject movement-mirroring avatar and motion-retargeting 3D hologramimage generator (604), and a subject-to-avatar recognition and posetracking block (605), as illustrated in the system block diagram (600)in FIG. 6. In the preferred embodiment of the invention, thesubject-to-avatar recognition and pose tracking block (605) isconfigured to receive subject feature extraction points, typicallythrough a camera installed on a mixed-reality headset worn by an MRenvironment user and/or viewer (614). The initial feature extractionsfrom captured image frame(s) of a physical target (615) may be performedby a subject feature extraction block (607), which is ananalog-to-digital image conversion and processing module configured toprovide the subject feature extraction points that uniquely anddigitally represent image information in the captured image frame(s).The subject feature extraction block (607), as illustrated in the systemblock diagram (600), can be incorporated into the mixed-reality headsetunit locally or into the application layer structure on top of the MRenvironment graphics generation and holographic 3D visualizationoperating system (602).

After receiving the subject feature extraction points from the subjectfeature extraction block (607), the subject-to-avatar recognition andpose tracking block (605) attempts to match the subject featureextraction points with a known image identification dataset in aphysical target image database. If the physical target (615) issuccessfully identified through the subject feature extraction pointscomparison with the known identification dataset, the subject-to-avatarrecognition and pose tracking block (605) inquires a physical targetmovement-mirroring avatar superimposition user apps and avatar database(608) to retrieve a particular 3D holographic avatar (613), which ispreviously selected by a user to encapsulate the physical target (615)in the mixed-reality environment. The physical target movement-mirroringavatar superimposition user apps and avatar database (608) may also beoperatively connected to app repository developer tools (609) thatenable a plurality of internal or third-party developers to providevarious physical target movement-mirroring avatar superimpositiondesktop or mobile apps and a diverse set of 3D graphical avatars for theuser to choose from. This process is also previously described inconjunction with STEP 302 in FIG. 3. If the physical target (615) is ahuman subject, the subject can make a 3D avatar selection directly (i.e.616) through a mobile application on the subject's smart phone, orthrough system control panel and devices (612) connected to the physicaltarget movement-mirroring avatar superimposition and visualizationcreation system, as shown in FIG. 6.

Once the user-chosen 3D holographic avatar (613) is retrieved as acorrect encapsulating “jumpsuit” match to the physical target (615), thesubject-to-avatar recognition and pose tracking block (605) also beginsto track motions and expressions of the physical target (615), andtransmit the dynamically-changing feature extraction points from thecaptured image frames to the subject-to-avatar pose matching andreal-time movement mirroring and retargeting engine (603), as shown inthe system block diagram (600). In the preferred embodiment of theinvention, the subject-to-avatar pose matching and real-time movementmirroring and retargeting engine (603) executes motion retargeting bycorrelating the real-time motion tracking points of the physical target(615) to the user-chosen 3D holographic avatar (613) to mirror and mimicthe movements and the expressions of the physical target (615) in theuser-chosen 3D holographic avatar (613).

In the preferred embodiment of the invention, the motion retargeting andmirroring performed by the subject-to-avatar pose matching and real-timemovement mirroring and retargeting engine (603) achieves additionalefficiency and processing speed by querying a pre-defined pose featuredatabase for avatars (611 in FIG. 6) and rapidly determining the mostappropriate changes in the avatar's pose based on the outputrecommendations from a deep-learning artificial intelligence (AI) poseestimation engine (610 in FIG. 6) for 3D holographic avatars. Forinstance, when the motion and expression-tracked physical target (615)moves or changes facial or bodily expressions, the changes in pose andmotion extraction point data for the tracked physical target (615)invoke a similar magnitude of pose and motion changes in the user-chosen3D holographic avatar (613). Such reflective and corresponding changesin motion and expressions in the user-chosen 3D holographic avatar (613)fundamentally “mirror” whatever the underlying physical target is doing,while the physical target still remains invisible and totallyencapsulated by the user-chosen 3D holographic avatar (613) in the eyesof the MR environment user and/or viewer (614).

Once the subject-to-avatar pose matching and real-time movementmirroring and retargeting engine (603) of the physical targetmovement-mirroring avatar superimposition and visualization creationsystem completes the motion retargeting synthesis for the user-chosen 3Dholographic avatar (613), a subject movement-mirroring avatar andmotion-retargeting 3D hologram image generator (604) utilizes thesynthesized retargeted motion data to generate one or more updatedmotion frames for the user-chosen 3D holographic avatar (613), aspreviously shown in STEP 306 of FIG. 5. Then, an avatar visualizer block(606) in the physical target movement-mirroring avatar superimpositionand visualization creation system can display the updated motion framesfor the user-chosen 3D holographic avatar (613) in the eyes of the MRenvironment user and/or viewer (614), when the mixed-reality environmentis active. In the preferred embodiment of the invention, when a humansubject or another physical target moves, the user-chosen 3D holographicavatar also moves within the mixed-reality physical space in equaldirection and magnitude to mirror the physical target's changes inmotion and location, while keeping the underlying physical targetencapsulated and invisible to the HMR viewer.

Furthermore, in one embodiment of the invention, a rapidthree-dimensional holographic model generation from a dedicatedreal-time model reconstruction studio with multiple camera angles may beutilized as a component of the physical target movement-mirroring avatarsuperimposition and visualization creation system for synthesizing 3Dholographic avatar models. The rapid 3D holographic model generation mayutilize the app repository developer tools (609) and the physical targetmovement-mirroring avatar superimposition user apps and avatar database(608), as shown in the system block diagram (600) in FIG. 6.

In a preferred embodiment of the invention, the dedicated real-timemodel reconstruction studio with multiple camera angles is called“HoloPortal.” HoloPortal is a 3D model creation studio with a real-timeor near real-time 3D reconstruction system. This 3D model creationstudio is configured to place a target object (e.g. a human, an animal,or another moving object) in a designated area of the 3D model creationstudio that positions a plurality of cameras in various angles aroundthe designated area to capture multi-angle video footages of the targetobject. Then, the multi-angle video footages are processed, transformed,and reconstructed as dynamic 3D models, which may include 3D meshmodels, textures for all related frames, and movement frames associatedwith the target object. After the dynamic 3D models, also called hereinas “3D body doubles” are generated from the HoloPortal, the dynamic 3Dmodels can be stored in a 3D holographic avatar database (e.g. 608 inFIG. 6).

Then, the physical target movement-mirroring avatar superimposition andvisualization creation system synthesizes a 3D mixed-reality artificiallayer, where one or more dynamic 3D models are selected and placed intospecific virtual coordinates next to locations of physical objects asmixed-reality objects (MROs) or mixed-reality holograms (MRHs), inaccordance with a mixed-reality designer's intended user interactionchoreographies. At least some of MROs and MRHs in such mixed-realityenvironments may be user-selected 3D holographic avatars configured tomirror the movements and the expressions of a physical target in realtime. Furthermore, the HoloPortal and the physical targetmovement-mirroring avatar superimposition and visualization creationsystem may be dynamically linked to an electronic social platform forsharing, monetizing, and viewing a variety of dynamic 3D models storedin the 3D model database. These dynamic 3D models may be generated in 3Dmodel formats such as OBJ's and/or COLLADA's.

In one example, HoloPortal first records multi-angle videos from amultiple number of cameras surrounding the designated area. Then, themulti-angle videos undergo silhouette extractions, 3D voxel generation,3D mesh generation, deformed 3D mesh generation, and texture-on-meshgeneration to create a 3D model, or a “3D body double” model through avariety of data transformations and graphics reconstructions executed ongraphics processing units incorporated in or associated with theHoloPortal. Preferably, the HoloPortal can generate 3D models (e.g. “3Dbody doubles”) and 3D contents in real-time or near real-time, withoutlengthy and laborious conventional methods of 3D content generationprocesses that can take many hours to many months. Furthermore, the 3Dmodels generated from the HoloPortal can be utilized in as characters ina mixed-reality application, an augmented reality application, a virtualreality application, a 3D computerized game, or a 3D animation movie.For example, a holographic 3D model (e.g. a “three-dimensional (3D) bodydouble” model created after the multi-angle video capture of a humanfigure) may be created and inserted into a mixed-reality artificiallayer correlated to a particular physical space in virtual coordinatesas a 3D holographic avatar, which is configured to mirror movements andexpressions of a physical target in real time. Moreover, in someembodiments of the invention, a computerized 3D model created from theHoloPortal may also be physically manufactured with a 3D printingmachine located within or outside the HoloPortal for commercial,promotional, business, or transactional purposes.

Furthermore, in some embodiments of the invention, the physical targetmovement-mirroring avatar superimposition and visualization creationsystem may utilize another component called “HoloCloud” for creation ofa three-dimensional holographic model, instead of or in combination withthe HoloPortal. The HoloCloud system provides a rapid three-dimensionalmodel generation process from uncalibrated multiple sources of videorecording of a targeted object and subsequent cloud computing-basedvideo data calibration and three-dimensional reconstruction of thetargeted object. Typically, the HoloCloud system comprises a pluralityof common consumer-level video recording devices (e.g. smartphones,camcorders, digital cameras, etc.) positioned in various anglessurrounding a target object (e.g. a human, an animal, a moving object,etc.), a scalable number of graphic processing units (GPU's) in ascalable cloud computing platform, a 3D pre-processing module, a 3Dreconstruction module, a background 3D graphics content, a 360-degreevirtual reality or video content, and a dynamic 3D model created by the3D reconstruction module.

The plurality of common consumer-level video recording devices generatea plurality of digitized video feeds (e.g. Video 1, Video 2, . . . Videon) in various angles for a target object, and then utilizes anintegrated or standalone wireless transceiver (e.g. a cellulartransceiver, a WiFi LAN transceiver, etc.) to transmit the plurality ofdigitized video feeds to a HoloCloud graphics processing unit (GPU) in acloud computing platform. In a preferred embodiment, the HoloCloud GPUincorporates a pre-processing module and a 3D reconstruction module. Thepre-processing module is configured to calibrate temporal, spatial, andphotometrical variables of the multi-angle digitized video feeds, and isalso able to generate background 3D geometry and 360-degree virtualreality video. The 3D reconstruction module is configured to providedepth map computations, voxel grid reconstructions, and deformed meshgenerations for eventual generation of dynamic 3D models. After numerousinternal stages of video extractions, transformations, andreconstruction through the HoloCloud GPU, the background 3D graphicscontent, the 360-degree virtual reality or video content, and thedynamic 3D models are electronically generated and subsequently utilizedas 3D figures, graphics, or holograms in mixed-reality applicationsrelated to the physical target movement-mirroring avatar superimpositionand visualization creation system.

Pre-processing and reconstruction procedures for the HoloCloud systemrequire cloud computing-based video data calibration andthree-dimensional reconstructions of a targeted object, in accordancewith an embodiment of the invention. A multiple number of commonconsumer-level video recording devices generates a plurality ofdigitized video feeds (e.g. Video 1, Video 2, . . . Video n) in variousangles for a target object, and then transmit the plurality of digitizedvideo feeds to a HoloCloud graphics processing unit (GPU) in a cloudcomputing platform. Typically, the cloud computing platform is acollective number of graphics computing machines that are dynamicallyscalable to deploy and assign a flexible number of GPU's for parallelprocessing, depending on the intensity of graphics computation,transformation, and reconstruction requirements for the plurality ofdigitized video feeds. For example, a larger number of GPU's may beassigned to perform 3D graphics processing if the plurality of digitizedvideo feeds has a high video feed count, long durations, and/or higherresolutions. In contrast, a smaller number of GPU's may be assigned toperform 3D graphics processing if the plurality of digitized video feedshas a low video feed count, short durations, and/or lower resolutions.

In cloud computing-based video data calibration and three-dimensionalreconstructions of the targeted object, each HoloCloud GPU canincorporate a pre-processing module and a 3D reconstruction module. Thepre-processing module executes calibration of temporal, spatial, andphotometrical variables of the multi-angle digitized video feeds, and isalso able to generate background 3D geometry and 360-degree virtualreality video. The 3D reconstruction module, on the other hand, performsdepth map computations, voxel grid reconstructions, and deformed meshgenerations for eventual generation of dynamic 3D models or characters.

After numerous internal stages of video extractions, transformations,and reconstruction through one or more HoloCloud GPU's that aretypically configured to scale and parallel-process a varying amount ofworkload for 3D content generation, the background 3D geometry graphicscontent, the 360-degree virtual reality video content, and the dynamic3D model are electronically generated and subsequently utilized as 3Dfigures or graphics in a mixed-reality application. The HoloCloudsystem, which utilizes a plurality of common consumer-level videorecording devices for multi-angle video feeds of a target object and ascalable number of HoloCloud GPU's for video extractions,transformations, and reconstruction of dynamic 3D models, enables casual(i.e. non-technical or not technically skillful) consumers to beprofessional-level 3D content creators or mixed-reality experiencedesigners, who are able to capture and generate 3D graphics contentsrapidly and inexpensively without necessitating specialized 3D contentrecording equipment and/or high-powered 3D graphics computing equipmenton site that are typically required in conventional 3D contentgeneration.

Furthermore, by wirelessly transmitting the recorded multi-angle videofeeds to a scalable number of HoloCloud GPU's executed in a cloudcomputing network that processes high-powered graphics computing tasksto generate dynamic 3D models, a casual content creator is not requiredto have an expert knowledge of 3D graphics pre-processing andreconstruction processes that may be electronically executed by athird-party HoloCloud service operator. Therefore, various embodimentsof the present invention enable convenient and pervasive casualuser-created dynamic 3D hologram models and 3D contents, which werepreviously difficult to generate with conventional 3D content generationsolutions.

In order to initiate creation of holographic contents for mixed-realityapplications, two methods of ubiquitous and rapid three-dimensionalmodel content generation and robust social sharing of holographiccontents by casual (i.e. non-graphics expert) consumers can be utilizedin accordance with various embodiments of the invention. A first methodof ubiquitous and rapid three-dimensional model content generationinvolves a dedicated 3D content generation studio (i.e. “HoloPortal”)that allows a casual consumer to walk into a HoloPortal facility tocapture multi-angle video feeds from professionally-installed multiplecameras surrounding a targeted area in the HoloPortal for a dynamic 3Dmodel generation from onsite graphics processing units. On the otherhand, a second method of ubiquitous and rapid three-dimensional modelcontent generation involves a plurality of consumer cameras at anylocation of a casual consumer's choice to capture multi-angle videofeeds around a target object, wherein the multi-angle video feeds aresubsequently transmitted to a cloud computing resource specializing in3D graphics processing to generate a dynamic 3D model. As describedpreviously, this second method of the dynamic 3D model generation iscalled “HoloCloud.”

Various embodiments of physical target movement-mirroring avatarsuperimposition and visualization creation systems operating in amixed-reality environment and related methods of operating such systemsdescribed herein provide significant advantages to conventionalaugmented reality, virtual reality, or mixed-reality applications. Forexample, an embodiment of the present invention provides a novelelectronic system and a related method of operation that enable moreadvanced levels of interactivity and visualizations in a mixed-realityenvironment. Examples of such advanced levels of interactivity andvisualizations in a mixed-reality environment include, but are notlimited to, virtualized switchover roles and images between hologramsand physical objects, and dynamic real-time rendering of holographicmotions superimposed on targeted physical objects.

Furthermore, another embodiment of the present invention provides anovel electronic system and a related method of operation thataccommodate physical target movement-mirroring avatar superimpositionand visualization in a mixed-reality environment for enhanced immersivemixed-reality contents and mixed-reality choreographic scenario designflexibilities.

In addition, an embodiment of the present invention provides a novelelectronic system that empowers a mixed-reality system user withelectronic user interfaces, physical target movement-mirroring avatarsuperimposition user apps and avatar databases, and app repositorydeveloper tools to create, select, or modify a preferred 3D holographicavatar as a holographically motion-mirrored and fully encapsulated andcomputerized visual representation of herself or himself for real-timeviewing by other users immersed in a mixed-reality environment.

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein.Accordingly, the scope of the invention should be limited only by theclaims.

What is claimed is:
 1. A method for creating physical targetmovement-mirroring three-dimensional (3D) holographic avatarsuperimposition and visualization in a mixed-reality environment, themethod comprising the steps of: choosing, via a system user applicationinterface, a 3D holographic avatar that visually encapsulates andreplaces a visual presence of a physical target wherein the 3Dholographic avatar becomes a movement-mirroring substitute for thephysical target at the physical target's current location while alsomaking the physical target completely invisible in the mixed-realityenvironment generated by a physical target movement-mirroring 3Dholographic avatar superimposition and visualization creation system;recognizing and correlating the physical target with the 3D holographicavatar by capturing an image of the physical target in a camera of amixed-reality headset worn by a holographic mixed-reality (HMR) viewer,extracting graphical feature points from the image, comparing thegraphical feature points to various 3D holographic avatars stored in aphysical target movement-mirroring avatar database, and finding acorrect match for a graphical similarity between the physical target andthe 3D holographic avatar previously chosen by a user to encapsulate andreplace the physical target in the mixed-reality environment; trackingmotions and poses of the physical target via a continuous imagecapturing from the camera on the mixed-reality headset and a real-timeextraction of moving graphical feature points of the physical target;retargeting the motions and the poses of the physical target to the 3Dholographic avatar by matching tracking points of similar body parts andfacial features between the physical target and the 3D holographicavatar, and by executing a deep-learning pose estimation engine thatretrieves a matching pose feature from a pre-defined pose featuredatabase for 3D holographic avatars; continuously matching the trackingpoints of similar body parts and facial features between the physicaltarget and the 3D holographic avatar in real time as long as thephysical target is making a movement or a change in facial or bodilyexpressions while being present in the mixed-reality environment;continuously executing motion retargeting, in real time, from thephysical target to the 3D holographic avatar to mirror and mimic themovement or the change in facial or bodily expressions; and from avisual perspective of the HMR viewer wearing the mixed-reality headsetor another viewing device, encapsulating and replacing the physicaltarget with the 3D holographic avatar as the movement-mirroringsubstitute for the physical target at the physical target's currentlocation while also making the physical target completely invisible inthe mixed-reality environment, wherein the 3D holographic avatar alsomirrors and mimics the motions and the poses of the physical target inreal time, whenever the mixed-reality environment is active.
 2. Themethod of claim 1, wherein the physical target is a human subject, ananimal, or an inanimate object physically present in a physical space ofthe mixed-reality environment.
 3. The method of claim 2, wherein theuser that chooses the 3D holographic avatar to encapsulate the physicaltarget is the human subject or the HMR viewer.
 4. The method of claim 1,wherein the physical target movement-mirroring 3D holographic avatarsuperimposition and visualization creation system comprises amixed-reality environment graphics generation hardware infrastructure, amixed-reality environment graphics generation and holographic 3Dvisualization operating system, a subject-to-avatar pose matching andreal-time movement mirroring and retargeting engine, a subject-to-avatarrecognition and pose tracking block, a subject feature extraction block,a subject movement-mirroring avatar and motion-retargeting 3D hologramimage generator, and an avatar visualizer block.
 5. The method of claim4, wherein the physical target movement-mirroring 3D holographic avatarsuperimposition and visualization creation system further comprises aphysical target movement-mirroring avatar superimposition userapplications and avatar database, the pre-defined pose feature databasefor 3D holographic avatars, the deep-learning pose estimation engine,and the mixed-reality headset.
 6. The method of claim 4, wherein themixed-reality environment graphics generation hardware infrastructurefurther comprises a scalable number of graphics processing units (GPUs),computer servers, and cloud computing resources to execute themixed-reality environment graphics generation and holographic 3Dvisualization operating system, the subject-to-avatar pose matching andreal-time movement mirroring and retargeting engine, thesubject-to-avatar recognition and pose tracking block, the subjectfeature extraction block, the subject movement-mirroring avatar andmotion-retargeting 3D hologram image generator, and the avatarvisualizer block.
 7. The method of claim 1, wherein the 3D holographicavatar is one of humanized figures, cartoon figures, animals, or nonlifeobjects.
 8. The method of claim 1, wherein the system user applicationinterface is generated by a desktop application executed in a personalcomputer, a mobile application executed in a mobile device, or awearable device application executed in the mixed-reality headset. 9.The method of claim 1, wherein the physical target movement-mirroringavatar database is operatively connected to a user application databaseand application repository developer tools.
 10. The method of claim 1,wherein the 3D holographic avatar that visually encapsulates thephysical target located in the mixed-reality environment operates as afull-body holographic jumpsuit that makes the physical target invisiblewhile retaining real-time motions and expressions originating from thephysical target, in the visual perspective of the HMR viewer wearing themixed-reality headset or another viewing device.